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ABSTRACT 

The stated purpose of this paper is to review some of 
the critical issues that arise in the planning and conduct of th,e 
evaluation of action programmes amongst the socially and 
educationally disadvantagedr and to suggest the means that might be 
adopted to overcome the inherent dif f.iculties. The discussion first 
focuses on the characteristics of social and educational action 
programmes. Thenr the functions of evaluative research are discussed. 
Following thiSr the discussion asserts that the coexistence of 
educational and social disadvantage in extreme forms tends to evoke 
programs which make either one or several interventions and assume 
that the interventions will make the social system work more 
effectively. It is advocated here, that in the planning stage^ a 
model be constructed into which the proposals for change can be 
fit ted and, through which, how they work and their intended effects 
can be seen. The value of such models, it is stated, is that they 
allow each proposal to be worked out in teirms of its specific 
objectives, the agencies of the syst^em and the instruments of action. 
The discussion then focuses on "the/Choice and formulation of 
objectives" and "the special status' of evolving programmes". Finally, 
the discussion highlights considerations regarding "the design of 
evaluation experiments" and "the relationship between research and 
action". 
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PREFACE 

The special difficulties which are involved in providing educa- 
tionally ana economically disadvantaged populations v/ith access to 
the fullest use of the facilities offered by the school, and thus 
promote equal educata.onal opportunity, have for some time been tackled 
by research and experiment in compensatory education. Parts of these 
problems have already been mentioned in the CERI publication "Equal 
Educational Opportunity" ( l) . A description and appraisal of selected 
U*iS. American programmes, "Strategies of Compensation: A Reviev; of 
rlducatioual Projects for the disadvantaged in the United States", (2) 
has been sponsored by C£RI and will be published at a later date. 

From its early beginning, CERI stressed the importance of the 
educational problems of disadvantaged populations and has for some 
years been collaborating with groups making research and experiments 
in pre-school education in the Netherlands, Korway, Sweden and the 
United Kin-^dcm. At an early stage of this work, it appeared to be of 
major importance to develop evaluation practices which would lead to 
an increasing; understanding of the value of experimental programraes 
in this field. 

The following , paper vxas written by Kr, M.A. Brimer, M.a,, School 
of j-ducation. University of Bristol, and is based on a close co-oper- 
ktion between the author and uhu members of the national groups. The 
author's conu id e rabies experience, gained through his work for the 
British E.P.A. projects, was an important advantage". 



J. R. GASS 
Director 

Centre for Educational Reseairch 
and Innovation 



( t) OHCD, i'^71. 
(2> OiiCI), ifi print. 



4 

- 5 - 



The purpose of this paper is to review some of the critical 
issues that arise in the planning and conduct of the evaluation of 
acti^on programm^s^^^ongst the socially and educutiunally disadvantaged 
and to suggest the m^a^is that mi»;ht be adopted to overcome the inher- 
ent difficulties. 

Before considering the applications of evaluation to action 
programmes one Qught first of .all to look at the concepts that are 
applicable to the programmes themselves. \ 



Thp* ^"^h^-rgot^^ri Cities of Sonigl gnd fid'jo >tlnn?il Actiron Programmes 



Such programmes generally accept as targets for rfeniedial action 
the forms oi' normatively identified deprivation which themselves help 
to define socially disadvantaged groups. The process woixLd be i:^tural 
and roper if there were not sometimes ambiguity over the status ^o^ 
the variables which identify the socially disadvantaged groups. 
Unfortunately, it is not always certain what is the innocuous and 
even cultiu^ally valuable context of disadvantage and what are the 
variables that must be modified. The consequences of this are that 
action programmes tend to be multiple and that^^rrsal hypotheses 
are often erected on the flimsiest of concomitant variation evidence. 
Moreover the multiple action is concerted so that there is little 
hope of being able to distinguish the particular consequences of any 
one actxcn or the interaction effects of the multiple variables that 
are bein^; rr.anipulated. 

Potentiality for confusion is aided to ambiguity in so far as 
the basis of both identification of such groups and the wish to 
institute action arises from a moral stand -point with ideological 
association. The concepts of equality or of equality of opportunity 
support the moral jud^^ement that action is necessary and introduces 
a notion of normative control. Objectives are defined in terms of 
normative standards; , the difference betweo^n sub-samples of the norma- 
tive ^^^roup being used to justify the need for particular forms of 
action related to the centrally tending targets. Discrepancies 
ob£:.erved at one point in time are not the result of short-term effects 
alone but oi'ten of generati'^ns of change. The long-term changes which 
have resulted in a particular group of people being in a less advan-- 
tageous position tnan others are often very diverse. Nevertheless 
the effect of different social and environmental influences is to 



bring the disadvantaged together into one apparently homogeneous 
group (either because of their common location or being regarded 
together because of their demands upon the community in a similar 
form e.g. intensive poverty) and lor their problems to be considered 
as if they arose from common caus».s and without regard to the his- 
torical ana biographical determinants of their current common dis- 
advantage • A further complicating factor arises when there is a 
, geographical concentration of such groups such that the interactiofi 
of their common symptoms of itself produpes phenomena which reinforces 
the social scientist's view of the common origin of their problems. 

Since the disadvantaged are commonly recognisable in terms of 
the absence or lower level ol certain attributes which characterise 
the normatively secure groups, it is a natural and rather naive step 
to then formulate compensatory programmes which seek to make good the 
deficiencies. So far as educational action is concerned it is agreed 
that the earlier children are treated the less likely it is that their 
domt^stic aeprivation will be allov/ed to become a progressive " handicap 
and it is not unnatural that the action should be modelled on what is 
considered to be characteristically good ^^ractice in normatively good 
domestic situations. Thus some of the commonest forms of action pro- 
/grammes are really attempts to simulate the conditions of a good home 
/in the more limited circumstances of an institutional pre-school 
setting. The naivety of such an approach consists not least in the 
failure to' perceive that the effects which are characteristic of good 
homes ana the actions Wuich are associated with them are the result 
of long-term exposure. It is not merely that the number of hours per 
day that a child si^ends in the context of a good home has its effect 
but also tnat there are regularities of cont.ext and personal inter- 
action which extend over a long^peripd usually unbroken. Attempts to 
simulate such characteristics will always be handijlcapped by the 
inability to reproauce both the intensity and the extensity of 
exposure. 

Awareness of the limitations of institutional action and of the 
danger o: the disadvantaged parental ,^enerauion exerting stronger 
counter influences, thereby p-.^rpetuatim: itself, provokes action to 
modify parental behaviour. But, the parents are sunpicious and re- 
sentful of the alien invasion of their family lil'e, so the community 
of whicn tney are part become o the target Cor action in the hope that 
its more compatible agencies can be ehlisted and will be more effective. 
Action becomeo wiaespr^ad, diverse and disconnected. 



By contrasib, to be successful, the choice .or particular torma 
of action must be dependent upon a clear perception of what is modi- 
fiable by what means, and what of that which is to be modified is 
regarded as instrumental in change and what is in itself an ultim- 
ately desired change effect. It is an obviou$ requirement that what- 
ever is seen tu be an instrumental change should be articulated with 
that which ic considered to be ^an ultimate effect. Moreover the haz- 
ard of constructing chains of instrumental change, with their multi- 
plied scope for breakdown, argues great caution and economy in their 
introduction, it may be preferable to devote more preparation and 
resources towards producing the ultimate effect directly. For example 
when it is considered that children in a certain ^:roup are incapable 
of using their first language adequately there may be a number of 
forms of both instrumental and ultimate effect considered as* worthy 
of achievement. It may be decided that one instrumental effect would 
be a change in the form of speech that' parents use towards their 
children. However, awareness of ^he difficulties or changing the 
speech habits of adults may lead to a decision to replace the inade- * 
quacies of t]ie parents by simulating the conditions of a good home 
in an institutional setting. l*'or thf- reasons already given such 
compensation will be unlikely to lead to change that would produce 
the desired effect, it may be preferable to consider what have 
formerly been regarded as symptoms rather tiian as causal factors and 
to change the speech characteristics o^ such children by intensive 
selective treatment. It is at this point that one iriay become aware 
^hat the changes which are introdujed in a child's speech by such 
directional effort are themselves cap xble of i>eing regarded merely 
as instrumental change. The relationship between lan^mage and tho\;ight 
has been instanced as th-i reason for seeking to impx'ove lan^;uage. If 
pro^Trammes are aimed merely at the s.vmptomatic differe^nces between 
chiidr*;?n from good ana bad homes the effect may be merely to chancre 
the partictJ.ar am imraeaiatC rasnion ol' speech of the^e children 
witfiOat havin^'" any lon,':-term effect upon the way of structuring 
roalit.v that the child has at his aisposai. An awareness of the dis- 
tinction uetAieen learning anj perf )rmance would be most important at 
this juncture as w»r.^ll as the r*^cognition that v/hut it is desired to 
produce in such children is a set of structures which are capable of 
transfer to aifferent sitiition;^ rathor than improvement of immediate 
compet^-nce in particular areas of experience. 

A different problem of social action prograraraes arises from the 
aefinition of the target population, permittin^^ as it does the dis- 
a-ivantaged rroupc to be considered in isolation from others. Apart 
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from the obvious recognition that no group of people ever lives in 
complete isolation from others of different status and that its 
aspirations and its achievement luotivation are never formed in iso- 
lation, there is the practical difficulty that effective forms of 
action might, if they were applied to non-clisadvantaged groups, also 
lead to their improvement > so that the relative position of the groups 
would then remain the feame* Once such a pot:3ibiilty is recognised 
zhe moral dilemma involved in normative definitioii social goals 
increases. 

There are related problems wuich tax the research design, amongst 
these being the problems of regression effect when action is directed- 
at those who in achievement are well below the mean of a r^^presenta- 
tive sample of the population. 

In choosing programmes for applioatxcn amongst disadvantaged 
groups th re' is a temptation to create new programmes that have not 
demonotratod their worth with respect to a more general population. 
Thus there is a tendency for programmes for the disadvantaged to be 
innovatory not merely with r§spect to the particular kinds of dis- 
advantage exhibited but also ?n the ijense of applying novel method- 
ologies, previously untested on any group, let alone the disadvantaged. 
Just because action tends to be multiple and to involve long term 
treatments it is expensive. It therefore behoves any innovator to 
justify to himself/ very closely the reason for applying a particular 
innovation vnich has not been previously demonstrated to have effect 
within a less restricted population. ^ 

It must be appai'ent from the foregoing that to derive research 
results from experimental action amongst the disadvantaged is 
dii'ficult. Indeed, it is understandable that research should tend 
to be fo>.'otten in the welter oi" action and that evaluation should 
be regarded as a means of justifying action. 

b^valuative i(e search 

Th functionc of evaluative research are rather closely definable 
and tney must be distin.^uished in the first place from those of mere 
evaluation, iivaluation of itself only presupposes the assigrvment of 
value to anytnin^^ and does not prescribe the need .for any scientific 
ri>sour in the process Of assigning: value. i:>valuative research on the 
other nana implies the use of a methodology, the statement of a set 
of premisses, the conformity with a set of rules and obedience to 
certain crit^^ria. Blrma.n (Wol) has listed the following main func- 
tions of evaluative research.* 
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!• To discover whether, and how well, objectives are being 
fulfilled* 

2. To determine the reasc. ^ for specific successes and failures, 
% To uncover the principles underlying a successful programme. 

4. To direct the course of experiments with techniques for 
increasing effectiveness. 

5. To lay the basis for further research on the reasons for the 
relative success of alternative techniques. 

6. To redefine Ihe means to be used for obtaining objectives, 
and to redefine subgoals in the light of research findings. 

Implicit in such a statement of function is the recognition that 
action and research must be planned together and that the business 
of evaluative research is as much concerned with the raeans as with 
the outcomes. It further requires" that context be specified so that 
it is possible to disentangle the implications of particular con- 
textual characteristics in their interaction with particular aspects 
of the programmes. This notion of context also has implicat.ions for 
sampling. 

If, for exarnpl*^, the target population is all urban children 
beiween the ages three and five, whose per capita family income 
falls below a given level, the context of their homes and of their 
cultural milieu is as important as age and income level for general- 
ising the outcomes or' the study. Unless each neighbourhood of each 
city is to be regarded as containing: an independent target popula- 
tion, there must be rega^'d for generalisable aspects of context. In 
other woi'ds, the tarfjet population must be stratified for whatever 
contextual characteristics are regarded as sufficiently important 
to ^*on3ider separately. There will be many more contextual character- 
istics wnich are unsuitable for stratification but whicii will be 
observable and measurable within the sample. Kuitiple interactions 
of those characteristics with each other ana wifch aspects of the 
prorTramiT.e will occur and are potentially vital in accounting for 
the events of the study. 

V/hen such interactions oetv/een context and progrfUjjne are being 
considered it becomes necessary to formulate a model for the^ existing 
interaction witnin the contextual s^ystem and to specify the hypotheses 
of the effective change upon such interaction. Unless this is done 
it becomes impossible to test the multiplicity of possible hypotheses 
that mip:ht have been erected when employing a probability model. 



Mtjltlrl^ Action Pro.o'rammes and i)vnamio Contpxt Programmes 



The coexistence of educational and social disadvantage in extreme 
forms tends to evoke multiple action programmes or generalised single 
action programmes in which it is supposed that the effect of the 
action will be to increase the output of the multiple elements of 
the social system through its own dynamics. Such programmes are 
characterised by multiple objectives and they assume the interaction 
of agencies of instrumental change which may have a combined effect 
upon the realisation of objectives. Optimism about the combined 
effects of change or of the organic growth of the single change 
frequently prevents the specification of the particular interactions 
which are anticipated and, perhaps more important, restricts the 
.perception of those forms of interaction which m^y be disadvantageous 
as well as advantageous to the overall achievement of goals. 

The only possiblt^ way of dealing with this situation is t^ set 
at the planning stage a model into which the proposals for change 
can be fitted and through which their mode of operation and their 
intended effects can be seen. If the model can reveal interaction 
between agencies such tinat the coinbined effect of changes can be 
estimated so much the better. V/her? educational disadvantage is the 
focal symptom under scrutiny, it is reasonable to project the pro- 
grammers objectives in the educational terms which are focally 
important, although it may be tha"^ for other purposes or for a 
furtner prescription of anotht^r part the model one might set up 
social objectives as boin^r inoCrumental in educational change. 
Normally a compari son, of the pro^Tramme unaer test with another 
pro^"ram.^rie waich i.s regarded as standard, or as differing in some 
controlled fashion, is considered to' be a vital part of the evalu- 
ation. It follows that the model should contain svifficient 
flexibility to enable both sets of objectives arising from the 
standard and from the experimental pro^*:ra\j'ie to be encompassed. A 
convenient model whioh illustrates thj way in which the system may 
be constructed at the plannin,^ stage is that derived from Carroll's 
li9o3) moaei of- tjchool learnin^^. It also inwlves the notion that 
aii learners are capable of achievinr mastery given sufficient time 
ana the:c^efore all but tlLe criterion variable are measured in terms 
of time for learnin^3. The element;: of trie model mL^ht oe defined 
as follows: 
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1. Perseverance - readiness of the learner to invest 

actively in learning 

brought about by (a) level of motivation 

(b) de;;ree of positive attitude* 
Measured tnruu£:h r mount of available time utilised. 

2. Aptitude of Learner - readiness of th.> learner for a 
- particular kind of learnin*: brought 

about by 

(a) level of general ability 

(b) knovrled2:e already possessed* 
Measured through time taken to learn under dptimal conditions* 

3. Time allowed for learning - created by 

(a) curriculum 

(b) the^ disposition of other circumstances 
which might limit or increase 
ourriculum time* 

Measured through time provision* 

4* quality of the Learning Situation - dependent upon 

(a) the effectiveness of the teacher 

(b) the provision and organisation of ^ 
build im^s and equipment* i 

I'leasured by time requirod Tor learnin'" in excess of jthat 
iikinimaiiy reqiwrod by aj^titude* 

. ^)* Skill mar>ter:/ - aciii ^vemnni of r;oal objective-, measured 
tiirourh criterion - referenced tesvo* 

(The inl^-ntion ncre ic: to ei^ca^e froM norm r-'ferencea tests 
wr.icn are inca|.abie^ of s.ecifyin^j: the extent to wnich the curriculum 
objectives have been achieved*) 

' The mciei is diar,r?iiri!i.ed as follow:: - lines show the links and 
arrows tne airection of effect* 
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Kotivatxon 



Attitude 



"N. 7 



Percoverance 



I Curriculum 



Tiitie I 
Allowea I 



General 
AbiUty 




Quality of 
Learning Situation 



Other 
CiroUiT.3tances 





^ 




Teachers' 
Effectiveness 




Buildings 

and 
Equipment 


/ 







Laoh one tof tne input elements to tne model is capable of 
furtner speoif ic'it : on in terrr»s Of its own system where it falls into 
tne centre of\the uyotem anvi so on to thv.» boandary the intended 
scheiiie. ^ 

?or exan^'jl^?* teacLinr effectiveness mi^:ht iti^elf becoi:;e the 
focus of a system^ prescription and the elements of tn;;: n.oael mi^ht 
oe defiiied as follows: 

1« Perse\rerance - identif ic ftioii with aims of project, 
i^eperiilent on (a) Motivaiion 

(b) Attitude * 
Measureu t'-ro^^^^n tixe devot'.'d to project. 

2. Aptituue - ao^lity to contritjufce towards uch Lavement of 
aims of project. 
I>t?pendorit on (a) General lev-?l of 'ibiiity 

\ Kh) ooeciflc skills ana Knowloaf:o. 
!ve i.-:ir^a''tnro time rcvv^ircd aoo/e that miniirially po^;*aible 
to inc-lcat<^? skills and knowlea^e. \ 



Time avaii'-it>ie for increasing Teacher lif fectivenes:^#> 
jepenaent on va) Teach-^r load 

U') Other cornmitrrients. 
u.'4rca tr*rox-'h time witnout ino:::caL.atie other coirtridtment . 



/ 
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4« Teaching Situation 

Dependent on (a) Organisational effectiveness 

(b) Buildings and equipment. 
Measured through timu required above that minimally possible 
tc inculcate skills and knowledge. 




General Level 
of ability 



Specific skills 
and knowledge 



TEACHER 
EFFECTIVENESS 







,Teaching 




Other- 


' Load 




Commitments 




Organisational 
Effectiveness 



Buildings & 
Equipment 



Gleai^ly the models can be articulated further but it is often 
preferable to leave them at as general a level as one can in the 
early planning stage in order to aid coherent scrutiny, 

» The value of such models is that they allow each proposal to be 
worked out xn terms of its specific objectives, the agencies of the 
system ani the instruments of action, making explicit i^itentions 
for action am hypotheses of effect • Juccessively more, detailed 
specifications are made witnin the model until a defined programme 
eme rges. 



ERLC 



The Choice and Formulation of ^b.icctives 

Th" introduction has already exposed the tendency of projects 
ajimea at the uit^^idvantared to elt.vate concomitant differences between 
good and poor homes as argu/nents both for action and for the objec- 
tives to be sought. While it is obviously true that objective's can 
only be identified through an appeal to a value system, the values 
themselves do not prescribe^trategies which enable objectives to^ ' 
be selected or their status to be recognised' as part of a causal 
cnain. The proliferation of objectives arising from an indiscriminate 
appeal to a value system can frustrate action, by diffusing its 
processes, and evaluation by substituting multiple vague hopes for 



13 



- 12 - 



e'xpM-ci.t., expectations. In seeking to explain the inadequacies "of 
action programmes, criticism has been strangely levelled at the 
limitation of "^tlfe^-^S^ms of action carried out. (See, for example, 
TtSaSd^J^TC).. Thus broader value systems are drawn in, which specify 
objectives whicTPhave^iir^viously been conceived to be peripheral to 
the action and its obaectrvbsTATgiM^nt on the relative 

priority status of values and the restrictions of'lFaTu&^r^ho^en 
rather than upon the structural relationship of the sources of 
variation which might be manipulated. The application of a ..odel 
such as that suggested in the last section \>rill help to locate ob- 
jectives ana to screen tuem for their structurstl and causal status 
in relation to one another. Since all action programmes imply the 
existence of causal relationohios it is necessary to a^ipeal to a 
boay or theory whxch is more general than the particular hypotheses 
under test. / 

It is highly likely that in such programmes, competing, or 
apparently competing, theories will emerge. In particular there may 
be seen to be a competition between' those theories which imply a 
sociological vit^w of the^ system and those which imply a psychological 
view. Reddiford G. (1969) has cogently argued the independence of 
sociological concepts from those of psychology. It follows from such 
a thesis that sociolo^^ical concepts are not themselves e>.plicable 
entirely in terms of psychological concepts and it is possible to 
entertain toth withxn the same model on a non-competitive footing. 
There are rather more serious quest! *ns as to whether sociological 
qoncepts and constructs can ever be causal in character. Thus it 
• may be necessary to specify some objectives in sociolo.^^ical terns 
but to tarn to psychological constructs foi* the purposes of securing 
chan'^e which the sociologist woula then recognise in terms of his 
own modes of testing social systems. Whatever the means chosen for 
reconciling the apparently corripetitive demand of 5>ocial and behav- 
ioural scientists it is vital that the form of reconciliation should 
not merely be one of tolerance of f onris of independent action which 
cannot be related. 

Strategically it is always helpful to set out the categories 
of ijltimate 2:oal before consideri^.-; intermediate goals even though 
acxion may be airected at intermediate and instrumental change. To 
leave ultimate goals unexamined Is to risk sui^stituting within-system 
Efficiency for pay-off from the system. The categories of ultimate 
(^oal which might be screened are those v^hlch are considered to be 
the^hange areas for action: such categories as knowledge, skills. 
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attitudes, social interaction and role percei-tion may be set up as 
possible groups of outcomes. Under each of these headings the major 
objectives are listed and tnen tiieir subordinate objectives formu- 
lated until t-he point is reached wiien exemplars in an operationalised 
form are written. It^ is a considerable aid to employ a system such 
as the Taxonomy of 'iiaucationai Objectives (Bloom, 1956), whenever 
^he objectives form part of an oraeriy universe. It not infrequently 
happens thaTi tu • objecti"/e:5 are r.ot so \ea2ily Sjecifiea within a 
hierarchic structure and t:'jo is :r.ost a^par^n*. v.xen it is a construct 
which identii'ies the area ir* wx.ich change is to be 3oujht. Constructs 
by tr.eir nature cam.ot ue airectly sa^.pltfd am car. only be monitored 
through iuairect cbsorvr-^tiono wnicit are inferenta^il aeductions from 
the construct tr.eory. 

An exaf'.ple of sacn a eonHj;ruct ;;ould be in teruii. of the ai*ea of 
attitujie referred to above. As a coi^struct, 'attitude* refers to the 
QOi^-^r^nt ^tr^^ctaro of acti(,u and reaction tendencies possessed by 
an inji*v' ^'-xl or a -troup of persons related to a particular object, 
the struG^^ro fcev:. - r«^latlvely stable and resistant to change. It 

coriw^if^er-'i to fe affectively rather tr.an co^nitively determined 
althO-*ifh capable of rati nal expi ic^H^tion and to be associatea with 
Deiiefs nel'Jt A.Mcii are inkier ^nient of veridical evidence. Attitudes ^ 
are tnerefore mcjt con;:..oniv ri^easured by invitinp; judgements of 
ae^^rees of p-rsonal confon. ity with verbal stateiuents expressing 
opinion aboit the oc.iect. To sanr.cle the universe of aspects of the 
object or tue j^uivei^^e oi verbal st -;terr.%nts ox" opinion about aspects 
of xt^^i ct }eot /co.Ii cle^iri/ ::.Gufijci'nt an a basis for testing 
tr.^^ cor.rvruct itc;':fif. Therefore sc;;i^i: ar*- conirtruoted w;*ich accord 
to rescOf-ses to sta^v:*.erit ccaie v^.^^iS wnz^u are hoiro Teneo-i^ and 
therC'fore ai;:tive. Sucn or>svrv'iti<^u of responiies is clearly in- 
iir-etjy r-^iatt^a to tne .joa.Jtruot ii--eii' am indi-^'iduals may v?i.ry 
ir. tneir responii>i to V''^>"bqi otale-nents ;;it/nout thereb ' im^lyinr^ a 
3hift xr. the cta^U3 of tx^^ir -tt ' tUvies. For example, it has been a 
:?oc' on ^ir, ;i>.." in .!.n-i«na trrH. lie attitudes of teachers towards 
ea^oatjio*-. kni cfaiir-n t^^nu hec^^ut- rr.ore conservative and tough- 
rrJnded after a perioo in teafji-in-' than tney were w:.en the teachers 
were st^>'icnts. Jl, ^h a sr.Ift m otc^ei ved respon.s^c coulu equaljly well 
re '•XL'iair.>:"i ny a^-co«iif.,' to tn*-- roa^norc a ';reat<=^^ tolerance of 
7-»rba] Jt.iterr.er*t of cojv.,^-rv iv^ i tou/ai~minded opinion as not 
necessarily imoJyin ■ conwervativo or touf >i-minaed action. The lesson 
to be le xn 1 fror.. t^ia it-^ tnat c -nfadence in the f iu. tuation ,of 
oroervatiOi' a u-pi/jn, 7iriar' >'i it. the construct under test can 
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only be engendered if other sources of variation in observation have 
been controlled* 

N 

The Special Status of Evolving Programmes 

A distinction shoula be recognised between evaluation applied 
to already institutionalised programmes, newly applied but previously 
tested programmes and untried evolving programmes • All too often 
these aspects are inextricably mixed up together. While the first 
two types of programmes have much in common the third is very differ- 
ent and demands special consideration. 

The characteristic of an evolving programme for the evaluator 
is Tihat it requires short term feed-back of information on the basis 
of which the programme will itself change. It also needs to create 
provision so that modified parts of the programine can be reintroduced 
in a different setting without ^ias, Suchman (1967) has diagrammed a 
list 'of procedures tnat are necessary. 



Value Formation 



Assessing the Effect 
of this Cxoal Operation 
(Prorram --^valuation) 



Patting Goal Activity 
into Opertition 
(Prorram Operation) 



Goal Setting (Objectives) 



Goal Measuring (Criteria) 



sldentifying Goal Activityi 
(Program Planning) 



According lo tnis model the procedure recirculates until a point 
r^acnei wr.ere the system can produce an exit, that is to say, when 
the av.o».*ssment of tne effect of the ^whole operation does not lead to 
a redefinition o^ values and prog±anune objectives but rauner point 
the way to an extension ox the objectives in an 'end-on* form* One /of 
tne main differences in the evaluation of an evolving programme i^' 
therefore that the assessment is related to decision-making about \ 
7'iluev> 'rxXi\ objectives as well as to pi'^ogramrrie efficiency. The req^uire- 
r.v?nt 0 snort-term feed-back of evaluation consequences imposes i 
rectriotion^ on the type of evaluation that can be carried out, /In 
reneral this means that forms of evaluation must be sought which 
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maximise validity but which are more tolerant of variation in 
reliability • Programme planners must work in a manner v;hich is not 
unlike that of curriculum builders in that they must seek to antici- 
pate decision points concernia5 alternative ways of conducting steps 
in the programme* i)valuation would then be closely geared to a 
decision between competing modes of conduct. Glaser (1967) has out- 
lined the steps which are likely to be necessary in a curriculm 
building exercise pursuing this model* Briefly, the model requires 
the hierarchic specification of the objectives in which intermediate 
objectives are expressed in terms of their dependence upon earlier, 
and in terms of tneir contribution to later, objectives; the speci- 
fication both of the achievement and the learning characteristics of 
the learner; the selection of the most promising learning situations 
and procedures for co^npetitive experiment and evaluation which is 
aimed at testing the goodness of the objectives, the relative success 
of the competing learning procedures and th-^ goodness of fit of the 
procedures to the individual learner *s characteristics. 

In evolving a programme the commitment is essentially long-term 
but the form of experiment is not itself long-term. Apart from the 
etnical and institutional problems o: pursuing novel procedures with 
the sajne ^^roup of subjects over a long time there is the question of 
the uneconomic use of time in the development of the programme if 
every stage must be dependent upon the prior administration of an 
'earlier stage. If objectives are appropriately operationalised and 
criteria exist v/hereby individuals can be pre-tested to determine 
their degree of satisfaction oz" these objectives it becomes possible 
to carry turougn a series of simultaneous experiments in which various 
steps of tr.e programme are independently tried and only at a later 
stage fitted together. The benefit of this method is not merely ^ 
economy of time but also that it prevents the pursuit of inadequate 
instrumental objectives if later occurring objectives have been found 
to be inaaequate. A further ben*^fit according to this model is that 
it prevents the specification of programmes which have some spurious 
or accidental connection with the achievement of objectives. It 
becomes ^oscAhle within such an experimentally developed system to 
specify a causal relationship between the elements of the programme 
and the objectives achieved. The benefit is not merely in terms of 
the validity of tne pro>"ramr.e but also in terms- of the degree of 
conviction tnat is possible ^xmoi^^sz those wno are responsible for 
mounting it. /' 
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The Dealgn of Evaluation Experiment 8 

It would be inappropriate to attempt a complete study of experi- 
mental design in so far as it applies to evaluation research in such 
a paper* There has indeed been no dearth of publications which set 
out the problems and processes clearly* Perhaps one of the reasons 
why the wealth of information available has been heeded so rarely 
is that discussion of experimental design is linked with statistical 
formulav- and mathematical j^robability which are not comprehensible 
to all those who mount evaluation of action programmes. It might be 
asKed why those who write such books have not themselves taken more 
trouble to make their influence felt directly in the construction of 
evaluative reseai^ch. The reason is probably that the expert in 
research design so often finds himself intolerant of the apparent 
untidiness of action programmes and his partners in the exercise 
despaii- of matching the elegance that the research design expert 
requires. Such mutual frustration is even less conduc|.ve to a com- 
patible partnership when evaluation is seen as a distant requirement 
of a ^e].ort" subsequent to the study and when the short tern feed-back 
to theVction direator is heavily biased in favour of functional 
viability of the action* 

Evaluative research designs can be chosen on the basis of a 
compromise between the demands oT exhaustive testing of hypotheses 
and practical feasibility. The nature of the compromise is indicated 
by the degree of exclusion oT alternative hypotheses that are mini- 
mally required in order to carry public conviction to be a point of 
extending the action to larger saiTiples of the population. It follows 
that the degree of pophlsti cation of the evaluative research must be 
iiubject to diacussion by the planning team and is not solelj within 
the competence of .the research worker. It is the duty of the research 
worker to reveal cleurly to" the team the consequences of failing to 
adopt a more sophisticated design. 

it should be recognised at the beginning of the planning of 
evaluative research that no experiment is ever completely free from 
error. The function of experim ntal design is to enable the error to 
be knov;n and to be kept witiAin the limits that the study will tolerate* 
The main categories of error that aesign seeks to control are: 

X 

1. Variation attributable to subjects. 

Variation improperly associated with treatment. 
5. Variation attributable to group-specific reaction to treatment* 
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Error of type 1 arises when the subjects who are beir^ subjected 
to the treatment are themselves a biased sample of the target popxila- 
tion» Errors of type 2 arise when the treatment under test is .asso- 
ciated with biased conditions which may be 'a"^ function: 

. - Ox the treatment itself, e.g. novelty: 

- of the physical cijrcumstances, e.g. the classroom; 

- of those who are instrumental in presenting the treatment, 
e.g. the teacher; 

\ 

- of the social conditions in which the experiment takes place, 
e.g. the climate of the school; 

- of the groups to, which the treatment under tenst is applied, 
e.g. the state olf restlessness of the groups. 

\ 

The third type of error arises when the treatment under test 
reacts with some prior biased condition of the groups to whom the 
treatment is applied, e.g. when the curriculum previously followed 
is particularly 'conducive to the treatment vinder test. 

The moie of. controlling for these errors is of two basic k:^nds. 
The object of both is to eliminate the effect of the error from the 
evaluation of the experimental trial. The first control mode is con- 
cerned v;ith the use of prior knowledge about the possible sources of 
variation such that they can be eliminated or counterbalanced. The 
second mode involves the principles of randomisation, the effect of 
which is to eliminate dirfdictional bias in the allocation of subjects 
to groups or of treatments to groups. Although randomisation avoids 
directional bias it nevertheless admits the possibility of bias 
arising by chance. The virtue of randomisation is that the variation 
which can arise from chance can be accountable in the tests of 
si(^nif icance. 

An excellent account of research design as it applies to evalua- 
tion studies is given in "Evaluative Research" by E.A. duchman, 
Russell Jage Foundation, Kew York, 1967, Chapter \6. It is unnecessary 
to cover the same ground again. One of the purposes of this paper 
is to suggest those designs which are likely td^ maximise outcome 
from evaluation under the characteristic conditions which surround 
compensatory programmes. Three sets of conditions are posited which 
in themselves are stages in the strategy oi* programme preparation 
and testing. 

\ 
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The first condition arises when a particular region or class 
of aeprivation is being considered for programe preparation. The 
main objective of this stage is to determine what forms of deprivation 
exist and with whax conditions it is linked, and further, what hypo- 
theses of causal relationship and of interaction are tenable. As such, 
it is a survey in which a representative seimple of the target popula- 
tion is studied and the resulting observations are subjected to multi- 
variate analyses. The techniques of analysis to be applied are such 
standard forms as those described by Hope,K., "Methods of Multi- 
variate Analysis", University of London Press, 1968. The particular 
techniques that apply to the examination of causal relationships in 
ncn-experinental data are relatively more recent in development 
(Blalock, h.H. 1963 and 1964; Juncan, O.J. 1966; Keeslin^, J.W. 1968; 
Tukey, J.W. 1954; -/old, H, ly56 and 19b7; ^'/right, S.. I934 and I960; 
Yee, A.h. ana Gage, N.-u. 1968). The plannin.<r of the survey requiri^s , 
the same care in specifying the conditions which are to be investi- 
gated as an experimental study would do. The outcome of the analyses 
should be the hypotheseis to be tested and the indicators or variables 
to be used. 

Kesults of such studies would naxura^^iy lead to the adoption of 
certain hypothet;es of change agencies and instructional routes which 
could then form the second stage in which experiments .;ould set out 
to test the hypotheses. The conditions associated with this stage 
are likely be characterised b./ tentativeness of hypotheses to be 
tested and perhaps 01 a number of a.lternativG h:/poxheses to be tested. 
It v;ould be futile unaer these conlitions to embark on a lon^-term 
gamble with the lives of so many children and so much effort. It is 
preferanie to focus strategy on decision makin^^3 about those routes 
and tno3e agencies whicn are most likely to be profitable? in long- 
t^vju 3n dy. It follow:: thu^ at Ihi^. Jta ,e tn^ j^-rlmentai work 
Ghojli be 3aort-j.erK, i^houij e-nploy limited tar,^:et populatiom:, em- 
ploy ^mail sajTiples am snoula a^:iin spek to maximise outcom while 
miramiain-^ error. The hoct of dlsturbW^; effects in educ.tlonal and 
sociul settmr;,: Ic r.ach that the iice L^iiouli not he unnecessarily 
loaded a^^ainst tne treatment unaer test. In many caseo the attempt 
to, maximise tne outeome of interaction of treatments with individual 
differences fruGtrates experiment by imposin;- impossible demands 
upon the con.-'.titution o:' groups, katchin^: techni-iuen, whether by 
individuals or by levels, are not only increasingly difficult to 
implement as the number of determinin,;^ variables and their levels 
increAses but zenas to diminish the sliie of tne sample;^ under test 
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Irrelevantly and to restrict the variance of any one variable and 
to mako the experiment dependent upon the premisses of the foreseen^ 
covariance when other unknown covariance may arise which is a func- 
tion partly of the restrictive effect of representing each variatle 
at each level of every other* 

The suggestion is that a number of simultaneous short-term 
experiments t>hould replace the attempt to mount the long-term experi- 
ment immediately. An example taken from the present author's own 
work (Brimer, M.A. 1967) may serve to illustrate the mode of tackling 
the problem. The objective of the experiment was to discover the 
relative effectiveness of five orthographies in learning the code 
of the traditional iSnglish orthography. The design chosen aimed at 
the control of difxerences in initial ability of children, difx'erences 
between schools and teachers, the effect of change and novelty and 
of the method of instruction. The children were selected at random 
from the aj[.propriate age group within each of five schools and were 
allocated at random to one of three groups of eight to be tau^^ht 
through a particular orthography. The children were also pre-tested 
on a measure of listening comprehension so that covariance methods 
could ultimately be used to increase the precision of the randomised 
design. Orthograchies were allocated to scnools at random under the 
condition that each Suhool should have three different orthographies 
and that each ortho.rraphy shoula be followed in three different schools. 
By tnis means it was hoped to control the initial ability of the 
children in association v;ith orti.ographies and eliminate school 
bias. In each ochooi one teacher was responsible for instruction 
with all three ortho^^raphies. The effect of thir was to control 
teacher bias t,o the same extent that school bias v;as controlled. A 
further control on teach','r effectiveness ana an equalising of the 
effect of jTiOvelty was sought by introducing a form of prof^rammed 
instruction involvin-: simple teaci.in,^ macnlnes. The method of in- 
struction also permitted a continuous record to be kej^»t of the per- 
foirmance of each of the groups at eacn i;tage of the learning. Thus 
it was ultimateJ./ pos. ible to compare not only the final achievement 
on' each orthography but also to compare the rates of change of per- 
formance across orthographies. Th; actual instruction period was no 
more th-m fourteen days, yet the data derived were such that adeqiiate 
discrirfiiriation between the orthographies could be measured. 

By strategically mounting a number of simultaneous studies of 
no greater luration than tr.is it woula be possible to solve a nx;unber 
or cruci'ii problems in relatively isolated conditions before fitting 
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the contributory processes together into an articul^ated programme. 
Such a process would be much less dangerous if it were subsequent 
to the survey described earlier in which causal patns had already 
been tested on a non-experimental basis. 

The conipletion of thio second sta^^e would then lead naturally 
to the thira stage of mounting th3 lonf^term study which would be 
set in real circumstances. In contrast to the second stage, the 
tar^ret populacion would be fixed bearinf^ in mind the desirability 
of being able to generalise beyond the confines of limi'^ed local 
circumstances and the samples woula be drawn in such a way that they 
represented the target population in its naturalistic environment. 
In most cases this would mean the accei,>tance into the study of 
intact groups without the capability of oeing able to randomise 
subjects in relation to groups • Covariance methods would be used 
to control subject-linked, group variation and treatments //ould be 
assigned randomly to the intact groups. Covariance methods would 
permit greater pr'.*cision to be acnieved in the experiment through 
control of residual variation betv/een randomised treatment blocks. 
Evaluation woold be carried out on group means and would discount 
individual variation within groups. If the treatment were such that 
it were to oe aajusted to intermediate characteristics of the learner 
then the nev/ly constituted groups would become the unit of analysis 
rather tnan th^ individual learner. The constitution of such groups 
for treatment and analysis purposes does not necessarily imply that , 
they are taugnt in the same physical location but rather that a 
comiTion attribute oi treatment is being evaluated for its consequent 
change effect upon criterion-referenced measures. 

The rielationshiD betueen Research and Action 

li is accoi-tod as almost inevitaoie in the evaluation of acti-^n 
pro^.^rahiies th-it r search workers ana action directors should be at I 
variance with one another. In part this has seemed to arise from 
the aifi'ercnt objectives wuicii tney eacr* pursue, in part through 
the ae.:3ree of inripartiality towards the action that they represent 
and in part throu,2:h tne degree of involvement in the action that 
they each concede. There is a furth^jr variation which is attributable 
to tneir ae^ree of involvement in context. The action workers are 
inevitably involved in the immediate day to day context of the ex- 
perimental situation whereas the rese^'^on worker is to a large extent 
insulated froir* it. The aifference also arises through the kind of 
competence that eacn of thorn has to oxfer# The success with which 
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action workers pursue the promotion of their course of action 
depends upon their success in persuasion and their success in 
involving others in the pursuit of the desirable outcome. They tend 
to be social scientists who are sympathetic to the client and who 
are enthused by the form of action they are promoting and who regard 
T;he intervention of measurement or evaluative procedures as unsym- 
pathetic and even as unnecessary. Such peo^.l^ are, on the whole, 
not inclined towards quantifying or measuring whereas the research 
worker is essentially bound up with the conformity of action with 
the prescribed design, with the maintenance of a rigorous schedule 
of assessment and ultimately with the desire to demonstrate the 
efficiency of the procedures and txieir impartiality rather than to 
demonstrate the success of the form of action. 

These differences are often exacerbated by the relationship 
within the project of the action workers and research workers in 
BTi organisational s.ense. Wot infrequently the, research workers are 
called-in in a consultant capacity and may not belong to the organ- 
isation whi.^h is promoting the action. Not unnaturally this tends 
towaras creating friction which is nevertheless still evident when 
it is a sinjle institution w:dch is promotintj action and monitoring 
the^ conseqiienees. The differences oetween the personnel v/ithin a 
single institution caay to even more institutionalised and extreme 
tnan those between iri:;titutions# Prior agreement on the degree of 
con."»traint wr^iiJi the evaluative research can be allowed to exercise 
is no guarantee of harniOniauG relalioni;hip# Such agreement is 
ucually gaanea at^ the point at wrdch ambitions ar*' high and diffi- 
culties are minimised. The- situation changes markedly as soon as 
tile fleldwork beginj to taice place ana the action workers iind them- 
jeives in oeVtain difficulties in promoting action when they are 
seQi.in the co-opemt i on of the ho:*t of agencies which r.elate to ari*^ 
institutioii and ihe co-operatinn of the members of the institution 
iti^elf. It is often ea:;ier for action woz'ker:. to accede to adapta- 
tionL- of the fom of action to institutional stresses than it is 
i'or r-ecenrc worker, to agree to similar cnanges. No one can pretend 
that any '^.ean;: exiots to eliminate all such diff icuities* One mode 
v;hic.. will reducH such sources of iriction requires plannin^: in 
G.>^cn aetail tn'it norn^ally it wo^ild oe comparable with the organisa- 
tion of a moon-Ian iinr:. The construction of detailed flow diagrams, 
p-^rr^onai action cnarts and fail-safe procedures woula then be sup- 
ported by pilot ii^tudie:- and siuulation exercises^ '/ery rarely is 
sufficient t'me :^:pent in ].l^>nninr for each procedures to be accom- 
plished. Frequently the source of oiirficulty 'lies in the way of 
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handling tne project v;here it is incuTnbent on those who are going 
to carry it through to begin it as quickly as possible in order to 
be able to work within the budget permitted. ^Ihere is nevertheless 
another source of premature action in the impatience of those who 
are connected with Vu: project to ^:et ahead with it rather than to 
plan tr. rough all the oteis that are necessary. 

The evaluative research worker can never be in an '-ntirely 
happy position; F.G. Caro (l969i in a recent paper claims that "the 
researcher walis cn a tight rope between an affiliation with power 
groups and the freedo:* to hear all sides without becoming involved 
in internal conflicts." His very attempt at impartiality may pose 
a further problem. C. Argyrjls (196O) has argued that a research 
..orker who is more airectli/ involved in a pro^p:ramn'ie is more effective 
than one who represses hi?^ humanitarianism in* favour of the demand 
for research. He even suggests that his attempt at impartiality 
leads to conflict with the action worker and that this conflict in 
its turn tends to render his observations untrustworthy. Ambiguities 
or .status and role, the conflicts between the demands of the research 
worker fo,r records and the tedium of record keepin^5 and jp:atherinf; , 
all impose constr^iints upon the research worker for wnich he must 
be prepared not merely in the sense oT bein^- v^illing to accet t such 
effects but ruBo in terms of planned procedures to deal with the 
consequences. 

Perhups it is in the budr:etary field that the research worker 
and action worker may fina tnemseJves most at variance. To the 
action worKer th^ expense of research may seem to be disproportionate 
to the a-TiO ^nt of effect it produces, liesearch information is often 
P'-rceived by the actiun worker as beinr inert and it is likely that 
those who are administering projects will look more sympathetically 
at iemands for further Dudgetary provision from those who are able 
to justif;; their requests on humanitarian ,f:ro-mcis rather than from 
those who require the money for evaluative purposes. It frequently 
na]-ir.ns that evaluation budgets are cut in oruer to supplement 
action. 
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Conclusion 

It is not possible i: a paper of these dimensions to emulate 
a handbook on evaluative research. The purpose of this paper has 
been to draw attention to the most profitable strategies and to 
refer to texts wnich are by their naoure rather more ae tailed than 
this paper could hope to be^ Reference has already been made to 
"Evaluative Research" by Ji^A. Suchman and another general text is in 
press which should complement it. The anticipated work is "The 
Evaluation of Instruction: issues and problems", edited by tUC, 
Vittrock and I>^^* V/iley Holt Rinehart, and Winston, 1970* 
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